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Description 

Field of The Invention 

The present invention is related to the field of digital 
image processing and more particularly to methods and 
associated apparatus for reducing quantization artifacts 
in a hierarchical image storage and retrieval system that 
employs the quantization of residual image components. 

Background Of The Invention 

Image pyramids provide an efficient means for the 
storage and retrieval of digital images at various resolu- 
tions. It is customary in such hierarchical image storage 
and retrieval systems to construct a multiple of image 
resolutions by repeatedly filtering and subsampling a 
high resolution image using a scheme similar to the 
teachings found in the patent to Melnychuck and Jones 
(U. S. Pat. No. 4,969,204). In such cases, the higher res- 
olution components are usually stored in the form of 
quantized residuals that are entropy encoded to save 
storage space. In the context of the present invention, 
we will make reference to one such system, namely the 
Kodak Photo CD system, as an example to demonstrate 
the fundamental concepts underlying the invention. 
However, it should be noted that this example is only 
used to give the reader an insight as to the operation of 
one or more embodiments of the invention with the un- 
derstanding that other hierarchical image storage and re- 
trieval systems with other resolutions or arrangements 
that store the high resolution components as residuals 
may be chosen to suit specific needs without detracting 
from the teachings of the present invention. 

The Kodak Photo CD system is composed of an im- 
age hierarchy or pyramid in which the highest resolution 
image contains 3072 x 2048 pixels and is referred to as 
the 16BASE image. This resolution is in most cases ad- 
equate to produce photographic quality originals on an 
appropriate digital output device. The next level of hier- 
archy, referred to as the 4BASE, is composed of 1 536 x 
1024 pixels and is adequate to generate a high quality 
HDTV display, or a small-sized photographic quality print 
on an appropriate digital output device. The lower reso- 
lution levels are, respectively, the BASE image consist- 
ing of 768 x 512 pixels (suitable for NTSC/PAUSECAM 
television viewing), the BASE/4 image consisting of 384 
x 256 pixels (suitable for sub-NTSC displays), and the 
BASE/1 6 consisting of 1 92 x 128 pixels (suitable for dis- 
playing thumbnail images). The complete image hierar- 
chy is constructed from the 16BASE image using the 
teachings found in the #204 patent to Melnychuck and 
Jones previously cited. The BASE/16, BASE/4, and the 
BASE images are stored onto a digital storage medium 
such as a CD in uncompressed form while the 4BASE 
and 16BASE images are represented as residuals and 
are quantized and compressed by using entropy coding 
prior to their storage on the CD. 



A quantizer is a many-to-one mapping and, as such, 
the quantized data often contains degradations com- 
pared to the original unquantized data. The degradations 
can be particularly significant when the input data has a 

s distribution (histogram) containing long tails, as those 
values are clipped to the outermost quantization level. In 
a hierarchical imaging system such as the Kodak Photo 
CD system, the residual image usually has a symmetric 
distribution with a high peak and decaying tails, some- 

10 times approximated by a Laplacian (two-sided exponen- 
tial) distribution. The minimum-mean-squared-error 
quantizer that accommodates such a distribution con- 
tains dense inner level (in the region where the signal 
peaks) and sparse outer level (around the tails). An ex- 

f5 ample of how a quantizer processes treats signal values 
is shown in the table of Fig. 10. It is based on the principal 
that by introducing a larger error in those values where 
the signal occurs infrequently and a smaller error at val- 
ues where it occurs more frequently, the overall average 

20 error can be minimized. 

As described in the teachings of the previously cited 
patent to Melnychuck and Jones, the construction of a 
high-resolution image is accomplished by adding the 
quantized residual to the interpolated low-resolution 

25 component from the previous stage. Thus, the error re- 
sulting from the quantization of the residual is propagat- 
ed to the higher-resolution images. Although the residu- 
als with large magnitudes occur infrequently, it should be 
noted that they often represent high -contrast edges or 

30 details in an image, and the introduction of a large quan- 
tization error can create an annoying visual artifact or 
loss of detail in the high-resolution image. One of the ob- 
jectives of the present invention is to introduce a method 
whereby the low-resolution image whose image quality 

35 is less critical can be modified in order to minimize the 
quantization errors in the high-resolution image. 

The aforementioned hierarchical storage technique 
is combinable with the insertion and removal of a digital 
watermark according to the teachings of 

40 EP-A-94420293.6 by Rabbani et al., to form a distributed 
image system. In a distributed image system it is com- 
mon to deliver an image of compromised image quality 
for the purposes of browsing or proofing. The compro- 
mise can be made with the use of a removable water- 

45 mark. Primary use of such a system is by professional 
portrait photographers who distribute a plurality of imag- 
es to a customer for selection and approval. The water- 
mark is a form of graphic overlay that may contain a cop- 
yright notice or information regarding the restricted use 

50 of the image. 

Upon selection of the desired image by the custom- 
er, the professional delivers a high quality rendition of the 
image either digitally or in hard copy form without the wa- 
termark. At all times the professional possesses the sole 

55 means of generating and delivering unmarked high qual- 
ity images. In a conventional photographic system the 
means for generating would be the original negatives of 
the images; in a digital hierarchical system according to 
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the invention of Rabbani et al., the means for generating 
are higher resolution residual components. 

In a digital imaging system, and in particular one that 
includes a hierarchical form of digital storage and retriev- 
al, the professional may use a suitable digital storage 
medium such as a CD for the distribution of proofs. In an 
unrestricted environment, the customer may choose a 
desired image resolution from the hierarchy for the pur- 
poses of browsing, proofing, or hard copy fulfillment. In 
those instances where it is desirable for the professional 
to deliver the digital storage medium containing the en- 
tire image hierarchy to the customer, it is also most eco- 
nomical to record the entire image hierarchy once onto 
the digital storage medium and avoid having to make a 
second copy containing only low resolution components 
for distribution. However, it is also desirable to restrict 
the use of selected high resolution components for the 
purpose of full image quality fulfillment until payment has 
been received. 

The invention of the Rabbani et al. application, pre- 
viously referenced, teaches a method whereby the pro- 
fessional places a digital watermark on a selected image 
component. The removal of the watermark is done 
through an additional image component containing the 
reverse of the watermark. The customer, having posses- 
sion of the CD storage medium, would possess the 
means tor generating his own high quality hard copy 
when authorized by the professional. 

When a watermark is applied to a low-resolution im- 
age, the pixel values in that region of the low-resolution 
image affected by the watermark can become complete- 
ly different from the pixel values in the high-resolution 
image corresponding to the same region. As a result, in 
those regions, the difference between the interpolated 
watermarked image and the high-resolution image can 
become fairly large which, given the nature of the quan- 
tizer, can result in large quantization errors. These errors 
often appear as a visual trace of the watermark in the 
high-resolution image and significantly impair the visual 
quality of the high-resolution image. Minimization of such 
errors is addressed by the present invention. 

Summary of the Invention 

One embodiment of the present invention is a meth- 
od for reducing quantization artifacts in a hierarchical im- 
age system of the type that decomposes a high resolu- 
tion image into low resolution image components, includ- 
ing low resolution image pixel values, and residual image 
pixel values comprising the steps of:: 

a) forming a predicted higher resolution image by 
interpolating the pixel values of the low resolution 
image components; 

b) determining the difference values between the 
pixel values in the low resolution image and the cor- 
responding pixel values in the higher resolution 
image; 



c) comparing each of the difference values against 
quantizer conversion values to determine if an error 
would exist for an executed conversion; 

d) adjusting the image pixel values within the low 
5 resolution image component if an error would exist, 

to a value that eliminates the error to form an 
adjusted low resolution image; and 

e) using the unadjusted image pixel values as part 
of the adjusted low resolution image component if 

10 an error does not exist. 

Another embodiment of the present invention is a 
method for reducing quantization artifacts in a hierarchi- 
cal image system of the type that decomposes a high 
15 resolution image into low resolution image pixel values 
and residual image pixel values and applies a watermark 
to the low resolution image pixel values comprising the 
steps of: 

20 a) applying watermark pixel values to pixel positions 
within the low resolution image to form a water- 
marked low resolution image component; 

b) determining the difference values between the 
pixel values in the watermarked low resolution 

25 image and the corresponding pixel values in the 
higher resolution image; 

c) comparing each of the difference values against 
quantizer conversion values to determine if an error 
would exist for an executed conversion; 

30 d) adjusting the watermark image pixel values within 
the watermarked low resolution image component if 
an error would exist, to a value that eliminates the 
error to form an adjusted watermarked low resolu- 
tion image; and 

35 e) using the watermark image pixel value as part of 
the adjusted watermarked low resolution image 
component if an error does not exist. 

From the foregoing it can be seen that it is a pre- 
40 f erred object of the present invention to provide a method 
and associated apparatus for improving the quality of a 
high-resolution image component by modifying a low- 
er-resolution image component whose image quality is 
less critical. 

45 it is another object of the present invention to im- 
prove the quality of a high-resolution image component 
by modifying a lower-resolution image component 
whose image quality is less critical when a watermark is 
inserted and removed. 

so Yet another object of the present invention is the re- 
duction of quantization artifacts in the addition and re- 
moval of a digital watermark to and from a selected res- 
olution image where the watermark removal record is 
placed in a higher resolution image component. 

55 The above and other objects of the present invention 
will become more apparent when taken in conjunction 
with the following description and drawings wherein like 
characters indicate like parts and which drawings form 
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a part of the present invention. 

Brief Description Of The Drawings 

Figure 1 illustrates, in block diagram form, a PRIOR 
ART hierarchical image decomposition scheme com- 
bined with a scheme for inserting a digital watermark in 
a low-resolution image component and placing a water- 
mark removal component in a high-resolution residual. 

Figure 2 illustrates, in block diagram form, a PRIOR 
ART hierarchical image reconstruction scheme for re- 
covering a high-resolution image that has been decom- 
posed by the scheme illustrated in Figure 1 . 

Figure 3 is an example of the brightness component 
of a BASE image in pixel value form. 

Figure 4 is an example of a watermarked BASE im- 
age in pixel value form. 

Figure 5 is an example of the pixel values of a 
4BASE image corresponding to the pixel values of the 
BASE image of Figure 3. 

Figure 6 illustrates the relative X.Y position of the 
pixels used in a linear interpolation scheme. 

Figure 7 illustrates the interpolated watermarked 
BASE image pixel values corresponding to the pixel val- 
ues of the watermarked BASE image of Figure 4. 

Figure 8 illustrates the pixel values of a residual im- 
age formed as the difference between the images in Fig- 
ures 5 and 7. 

Figure 9 illustrates the pixel values of the residual 
image of Figure 8 after quantization. 

Figure 10 illustrates, in table form, the quantizer in- 
put-output characteristics used in generating the quan- 
tized residual image pixel values of Figure 9. 

Figure 11 illustrates the pixel values of the recon- 
structed 4BASE image using the PRIOR ART schemes 
of Figures 1 and 2. 

Figure 12 illustrates the improvement to the PRIOR 
ART system of Figure 1. 

Figure 13 illustrates, in a flow diagram form, the 
steps necessary for the adjustment of the values of the 
low-resolution image pixels during the first pass of the 
present invention. 

Figure 14 illustrates the pixel values of a modified 
watermarked BASE image resulting from a first pass of 
the present invention. 

Figure 1 5 illustrates the pixel values of a reconstruct- 
ed 4BASE image using the pixel values of the modified 
watermarked BASE image of Figure 14. 

Figure 16 is a flow diagram of the procedure em- 
ployed in STEP 1 of the second pass of the present in- 
vention to adjust the pixel values of the low-resolution 
image. 

Figure 17 is a flow diagram of the procedure em- 
ployed in STEP 2 of the second pass of the present in- 
vention to adjust the pixel values of the low-resolution 
image. 

Figure 18 is a flow diagram of the procedure em- 
ployed in STEP 3 of the second pass of the present in- 



vention to adjust the pixel values of the low-resolution 
image. 

Figure 19 illustrates the pixel values of a modified 
watermarked BASE image resulting from a second pass 
5 of the present invention. 

Figure 20 illustrates the pixel values of a reconstruct- 
ed 4BASE image using the watermarked BASE image 
of Figure 19. 

10 Detailed Description Of The Invention 

The present invention generally addresses the po- 
tential problem of artifact generation in a high resolution 
image caused by the quantization of residuals. One ex- 

15 ample is when the high resolution image contains high 
contrast edges or areas of detail that can give rise to a 
large residual value. Consequently, the quantization or 
clipping of this residual value can degrade the quality of 
the high resolution image reconstructed from this resid- 

20 ual. 

Another example is when a watermark is inserted 
into the low resolution image. In this case, the pixel val- 
ues in that region of the low-resolution image affected by 
the watermark can become completely different from the 

25 pixel values in the high-resolution image corresponding 
to the same region, thus resulting in large residual val- 
ues. The error resulting from the quantization of such 
large residual values often appears as a visual trace of 
the watermark in the high-resolution image. 

30 A PRIOR ART watermark system arrangement is il- 
lustrated in Figure 1 . This arrangement is the same as 
the arrangement illustrated in Figure 2 of US. Patent Ap- 
plication Serial No. 146,371, referenced above. In Box 
34, a watermark W is first inserted in the BASE image I 

35 to arrive at a watermarked BASE image l w . This water- 
marked BASE image is then interpolated to the size of 
the 4BASE image in interpolator 24 using linear interpo- 
lation. A difference is formed between the original 
4BASE image taken from the output of a decomposition 

40 apparatus 101 and the interpolated watermarked BASE 
image in subtracter 32 to form a modified 4BASE resid- 
ual that serves as the watermark removal record. This 
modified 4BASE residual is encoded by encoder 20, en- 
crypted in encryption unit 26 and stored on a CD storage 

45 medium 10 along with the watermarked BASE image l w 
in unencrypted form in addition to the original unencrypt- 
ed BASE/4 and BASE/16 images. Additionally, a 16 
BASE residual is formed and added to the CD storage 
medium 10. Note that except for a negligible increase in 

so the size of the compressed 4BASE residual, there are 
no additional memory requirements since the watermark 
removal component is embedded in the 4BASE residual. 

Throughout this description of the invention refer- 
ence is made to the forming, merging, or differencing of 

55 images. It is understood by those skilled in the art that 
the pixel values representing images are manipulated, 
not the images themselves. 

For browsing or proofing, the PRIOR ART system of 



4 



7 



EP 0 705 025 A2 



8 



Figure 2 is employed. The PRIOR ART system arrange- 
ment of Figure 2 is substantially the same as the arrange- 
ment illustrated in Figure 3 of US. Patent Application Se- 
rial No. 146,371, referenced above. The user retrieves 
the BASE/1 6, BASE/4, or watermarked BASE images di- 
rectly without decryption from the CD storage medium 
10. Upon authorization, the user inputs a decryption key 
to the data decryption unit 28 to decrypt the modified 
4BASE residual. The watermarked BASE image is de- 
coded in decoder 30 and interpolated using linear inter- 
polation and is added to the decrypted modified 4BASE 
residual in the reconstruction apparatus 201 to recover 
the 4BASE image. Details of the reconstruction appara- 
tus 201 can be obtained from Figure 2, and the associ- 
ated description found in US. Patent No. 4,969,204. In 
general, if the modified 4BASE residual is quantized prior 
to its storage on the CD 10 (which is usually the case), 
the exact 4BASE image cannot be recovered. The dis- 
crepancy between the original 4BASE image and the 
4BASE image recovered according to the above proce- 
dure depends on the coarseness of the quantizer in ad- 
dition to the nature of the watermark. The quantizers 
used in the PRIOR ART systems of Figures 1 and 2 are 
part of the encoders 20 

The description of the problem encountered with the 
PRIOR ART systems of Figures 1 and 2 can be clarified 
by using an example. Assuming that an image is repre- 
sented by a two-dimensional array of pixels with the pix- 
els being represented by numbers with indices (i,j) de- 
noting the location of a pixel at the ith row and the jth 
column, where i=l,...,N|, andjd,...,N pi and^ and N p are, 
respectively, the number of the lines and the number of 
pixels per line in the image, and that the brightness of 
each pixel is represented by an 8-bit number, i.e., a 
number in the range of 0 to 255. Now consider an exam- 
ple where, for the purpose of simplicity, a low-resolution 
image, hereafter referred to as the BASE image, is com- 
posed of a 5x5 array of pixels as illustrated in Figure 3. 
Referring to Box 34 in Figure 1 , a watermark W is added 
to this image to create the watermarked BASE image of 
Figure 4. The continuous line in Figure 4 shows the re- 
gion affected by the watermark. In this example, the ef- 
fect of the watermark is to replace the value of the pixels 
overlaid by the watermark W by a uniform brightness val- 
ue of 200 so as to create a high contrast between the 
watermark area and the remaining pixels in the BASE 
image. 

Figure 5, hereafter referred to as the 4BASE image, 
shows an image that consists of a 9x9 array of pixels 
corresponding to the high resolution rendition of Figure 
3. Referring to Box 24 in Figure 1, the watermarked 
BASE image in Figure 4 is interpolated to create a pre- 
diction for the 4BASE image. The interpolation scheme 
used in this example is linear interpolation and is de- 
scribed with reference to Figure 6. For best system per- 
formance the interpolator 24 should match the interpo- 
lator 49 (shown in Figures 16, 1 7, and 1 8). The four pixels 
in the low-resolution image labeled X, through X 4 are 



linearly interpolated to create the pixels Yj through Y 3 
according to the following relationships: 

Y,= (X 1+ X 2 )/2, 
Y 2 = (X |+ X 3 )/2, 

Y 3 =(X 1+ X 2+ X3 + X 4 y4, 

where, in order to insure that the Y s values are also 8-bit 

10 integers, all fractions are rounded to the nearest integer 
and a fractional value of 0.5 is truncated down, e.g., a 
value of 58.50 is truncated to the integer value of 58. Fig- 
ure 7 illustrates the interpolated watermarked BASE im- 
age according to the above interpolation scheme where 

is the boxes highlight the pixels that correspond to the 
low- resolution image (watermarked BASE image) of Fig- 
ure 4. Referring to the subtracter 32, in Figure 1 , the mod- 
ified 4BASE residual image is formed as the pixel-by-pix- 
el difference of the 4BASE image of Figure 5 and the 

20 interpolated watermarked BASE image of Figure 7. Fig- 
ure 8 illustrates this residual image which is the input to 
Box 20 in Figure 1 . It can be easily seen that if the resid- 
ual image of Figure 8 is stored on the CD storage medi- 
um 10 without any quantization, the original 4BASE im- 

25 age can be completely recovered. However, in most ap- 
plications, to save storage space, the residual image is 
quantized prior to its storage. 

An example of the characteristics of a quantizer 
used in conjunction with the Kodak Photo CD system is 

30 shown in Figure 10. For this example, if the input to the 
quantizer is any number in the range of -47 to -255, it will 
be quantized (clipped) to the value of -48. Using the 
quantizer in Figure 10 to quantize the residual image of 
Figure 8 results in the quantized residual shown in Figure 

35 9 which is then entropy encoded (e.g., Huffman encod- 
ed) and stored on the CD storage medium 1 0. As shown 
in Figure 2, the 4BASE image is recovered by adding the 
quantized residual of Figure 9 to the interpolated water- 
marked BASE image of Figure 7. This results in the re- 

40 constructed 4BASE image shown in Figure 11. Due to 
the errors introduced by the quantizer, the 4BASE image 
reconstructed in this way is significantly different from the 
original 4BASE image especially in the regions affected 
by the watermark. The underlined pixels in Figure 1 1 are 

45 those that substantially differ from the original 4BASE im- 
age of Figure 5. The standard deviation of the pix- 
el-by-pixel error between the original 4BASE image and 
the reconstructed 4BASE image, also referred to as the 
root-mean-squared-error (RMSE), is computed to be 

50 63.89. A large portion of this error results from the fact 
that the quantizer clips the value of any negative residual 
with a magnitude larger than 48. If this reconstructed 
4BASE image is printed with a hard copy device or 
viewed on a monitor, it would display a visible trace of 

55 the watermark. 

Referring to Figure 1 2, in accordance with the teach- 
ings of the present invention, the watermarked low-res- 
olution image l w at the output of Box 34 is modified in an 
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image modifying unit 36 prior to its interpolation and stor- 
age on the CD storage medium in such a way as to re- 
duce the artifacts that result from the quantization of the 
modified residual in the encoder unit 20. The extent of 
the modification depends on the characteristics of the 
quantizer and the watermark. For example, for a fine 
quantizer which contains many quantization levels, the 
changes in the visual quality of the watermarked BASE 
image, as a result of the present invention, will be negli- 
gible. On the other hand, if a coarse quantizer with a high 
degree of clipping is used along with a high-contrast wa- 
termark, the changes will be more perceptible and will 
generally appear as a reduction in the watermark con- 
trast. However, it should be noted that in almost all ap- 
plications of interest, the benefits of recovering a high 
quality high-resolution image far outweigh any disadvan- 
tages of reducing the watermark contrast. 

The present invention introduces a method whereby 
the pixel values in a low-resolution image are modified 
prior to their encoding and storage in order to reduce the 
quantization artifacts in the higher-resolution image. The 
objective is to introduce the smallest possible magnitude 
changes in the low-resolution image pixel values that will 
also minimize the quantization error in the residual image 
while maintaining the viewability of the low-resolution im- 
age. In this way, the fidelity of the reconstructed high-res- 
olution image can be improved while minimizing any 
compromises to the image quality of the low-resolution 
image. In one embodiment of the present invention, this 
technique can be used to improve the reconstruction 
quality of a high-resolution image in a hierarchical image 
storage and retrieval system at the expense of introduc- 
ing minor changes in the low-resolution image. This is 
particularly useful for those applications where the image 
quality of a higher resolution image component is more 
critical than the image quality of the lower resolution im- 
age component. 

Referring to the flow diagram of Figure 13, which il- 
lustrates the functions performed by the image modifying 
unit 36, the difference (residual) between the value of 
each pixel in the low-resolution image and the value of 
its corresponding pixel in the high-resolution image is 
found. In the Photo CD example, where a high-resolution 
image has twice as many pixels in each dimension as 
the low-resolution image, the high-resolution pixel loca- 
tion corresponding to the low- resolution pixel location 
(i,j) is (2i-1 ,2j-l) and the difference is found as the output 
of subtraction operation performed at circle 50. The re- 
sidual value (difference) from circle 50 is then passed 
through quantizer 52 to output a quantized residual val- 
ue. The functional characteristics of a quantizer are 
shown in the chart of Figure 10. This value is then com- 
pared, in circle 60, against the residual value from circle 
50 to determine if an error exists. If it does the low-res- 
olution pixel value is adjusted at function 58, to make the 
quantization error zero. If no error exists no adjustment 
is required as stated in block 56. Each low resolution pix- 
el value from 56 or 58 is stored in a storage buffer 70 



pending the adjustment of all pixels forming the image. 
If a second pass is to be implemented it uses the pixel 
values stored in storage buffer 70. If a second pass is 
not to be implemented the pixels from storage buffer 70 

5 may be output to the CD 1 0 and to the interpolator 24. 
Carrying further the example discussed previously, 
the residuals between the pixels in the watermarked 
BASE image of Figure 4 and their corresponding pixels 
in the 4BASE image of Figure 5 are found and adjusted 

10 according to the above scheme. The result after a first 
pass is shown in Figure 14. For example, the result of 
subtracting the pixel value in location (1,1) of Figure 4 
from its corresponding pixel value in location (1 ,1 ) of Fig- 
ure 5 is +1 . Referring to the quantizer characteristics of 

is Figure 10, a residual value of +1 is quantized to the value 
+1 resulting in no error and thus no adjustment for that 
pixel. As another example, the result of subtracting the 
pixel value in location (4,3) of Figure 4 from its corre- 
sponding pixel value in location (7,5) of Figure 5 is -132 

20 which is quantized to the value -48. The smallest mag- 
nitude adjustment to the low-resolution pixel value that 
will reduce the quantization error to zero is -84. Thus the 
pixel value in location (4,3) of Figure 4 is changed from 
200 to 116. As is seen from Figure 14 the effect of this 

25 process is to reduce the watermark contrast in the wa- 
termarked BASE image. However, the quality of the re- 
constructed high-resolution image is significantly im- 
proved. 

Figure 15 illustrates the reconstructed 4BASE tm- 
30 age corresponding to the modified watermarked BASE 
image of Figure 14. Again, the underlined pixels differ 
from the original 4BASE image as a result of the quan- 
tization error. The RMSE between the original 4BASE 
image and the reconstructed 4BASE image of Figure 15 
35 is 3.09. This constitutes a significant improvement com- 
pared to the RMSE of 63.89 corresponding to the recon- 
structed 4BASE image of Figure 11 . As is seen from Fig- 
ure 15, a large portion of this error is due to the three 
pixels in locations (2,7), (2,8), and (2,9), which have been 
40 highlighted in a box. In the present invention, in order to 
further reduce the quantization error, the low resolution 
pixel values are adjusted in a second pass which con- 
sists of three steps as outlined below. In describing the 
various steps of the second pass, reference is made to 
45 Figure 6 which illustrates a neighborhood containing the 
four low-resolution pixels labeled X-, through X4 and the 
resulting linearly interpolated pixels through Y 3 . 

Starting from the top-left pixel of the low-resolution 
image resulting from the first pass stored in storage buff- 
so er 70, and scanning the image from left to right and from 
top to bottom in a raster fashion, STEP(S) 1 , 2 and 3 
described below are applied in succession to every pixel 
in the low-resolution image with the latest adjustment 
overwriting the previous pixel value in the storage buffer 
55 70: 

STEP 1 : For a given low- resolution pixel value X lt 
the interpolated pixel value Y 1 is computed according to 
the interpolation equation and its corresponding residual 
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value is found. The value of the low-resolution pixel X 2 
(to the right of ) is then adjusted by the smallest mag- 
nitude needed to eliminate the quantization error of that 
residual. Note that as a result of this adjustment, the re- 
sidual may sometimes be quantized to a value different 
from what it was quantized to prior to the adjustment. 
This is also true for STEP(S) 2 and 3 of the algorithm. A 
flow diagram describing the procedure employed in 
STEP 1 is depicted in Figure 16. 

STEP 2: For the same low-resolution pixel value X, 
as in STEP 1 , the interpolated pixel value Y 2 is computed 
according to the interpolation equation and its corre- 
sponding residual value is found. The value of the low 
resolution pixel X 3 (below X-,) is then adjusted by the 
smallest magnitude needed to eliminate the quantization 
error of that residual. A flow diagram describing the pro- 
cedure employed in STEP 2 is depicted in Figure 17. 

STEP 3: For the same low-resolution pixel value X., 
as in STEP 1 , the interpolated pixel value Y 3 is computed 
according to the interpolation equation and its corre- 
sponding residual value is found. The value of the low 
resolution pixel X 4 (to the right and below of X-,) is then 
adjusted by the smallest magnitude needed to eliminate 
the quantization error of that residual. A flow diagram de- 
scribing the procedure employed in STEP 3 is depicted 
in Figure 16. 

It should be noted that at any step of the second 
pass, the calculations are based on the modified low-res- 
olution pixel values that reflect all the adjustments up to 
that stage. Also, each pixel in the low-resolution image 
may be adjusted up to three times during the second 
pass. For those pixels that are located on the image bor- 
ders, only a part of the neighborhood as defined in Figure 
6 might exist. In such cases, only those steps for which 
a neighborhood is defined are carried out. The modified 
low-resolution image after the completion of the second 
pass would usually generate a very high-quality recon- 
struction of the high-resolution image. 

The application of the second pass of the present 
invention to the numerical example discussed before will 
help to illustrate further the operation of each step. Scan- 
ning the image of Figure 14 in a raster fashion, it is found 
that no adjustment is necessary until the pixel location 
(1 ,3) is reached. Denoting the value of this pixel by X 1t 
the value of the interpolated pixel Y-, in STEP 1 is found 
to be 92. The corresponding high-resolution pixel in Fig- 
ure 5 is located at (1 ,6) and has a value of 70. The re- 
sidual is thus -22 which results in no quantization error 
based on the quantizer characteristics of Figure 10. Sim- 
ilarly, the residual, corresponding to the interpolated val- 
ue Y 2 in STEP 2, is +1 and results in no quantization 
error. However, the value of Y 3 in STEP 3 is found to be 
92, which, once subtracted from the value 75 located at 
position (2,6) in Figure 5, results in a residual of -17. This 
residual is quantized to a value of -16, introducing a 
quantization error of -1 . The smallest magnitude adjust- 
ment needed to eliminate this quantization error will ad- 
just the pixel value X 3 , located at (2,4) in Figure 14, to 



119 instead of its current value of 118. After the adjust- 
ment, the residual value becomes -16 which is quantized 
without error. This procedure is continued until all the low 
resolution pixel values are processed. In this example, 
5 the low-resolution pixels that are located at (3,5), (4,2), 
(4,5), and (5,5), are adjusted three times during the sec- 
ond pass. 

Proceeding with the example, the completion of the 
second pass results in the modified watermarked BASE 
image of Figure 19. Note that while most pixel values in 
Figure 1 9 have undergone minor adjustments compared 
to their corresponding values in Figure 1 4, the two pixels 
in locations (2,4) and (2,5) have significantly changed. 
This was necessary to reduce the large quantization er- 
ror on the three pixels in the image of Figure 15 that are 
enclosed in a box. Figure 20 shows the reconstructed 
4BASE image corresponding to the modified water- 
marked BASE image of Figure 19. Again, the underlined 
pixels differ from the original 4BASE image as a result 
of the quantization error. It can be seen that all the pixel 
values in Figure 20 are within one unit or less of their 
corresponding value in the original 4BASE image of Fig- 
ure 5 except for the pixel located at (9,9), which shows 
a difference of 2 units. The RMSE between the original 
4BASE image and the reconstructed 4BASE image of 
Figure 20 is only 0.54, indicating the significant improve- 
ment resulting from the second pass. 

The high-resolution image, resulting from the 
present invention (Figure 20), compared with the PRIOR 
ART image of Figure 11, represents a significant im- 
provement. The price paid for this significant improve- 
ment is the replacement of the watermarked BASE im- 
age of Figure 4 with the modified watermarked BASE im- 
age of Figure 19 which has a reduced watermark con- 
trast. In most practical applications, the slight compro- 
mise in the usefulness of the low-resolution image com- 
ponent is far outweighed by the image quality improve- 
ments in the high-resolution image. 

The two-pass algorithm described above constitutes 
a preferred method for achieving the objectives of the 
present invention, namely, finding the smallest possible 
magnitude changes in the low-resolution image pixel val- 
ues that also aim at minimizing the quantization error in 
the residual image. It should be noted that several vari- 
ations to this main theme are possible. In general, based 
on simulations with the Photo CD image storage and re- 
trieval system, it was found that the preferred method 
resulted in a slightly better high- resolution image quality 
(for the same degree of modification in the low-resolution 
image) compared to the other variations. However, in 
some applications, other system considerations might 
make the use of the alternative approaches more attrac- 
tive. In what follows, a few of these variations are briefly 
described with reference to the preferred method. 

In one variation, only the first pass of the algorithm, 
as described above, is implemented to create a reason- 
ably high-quality, high-resolution image and the second 
pass is omitted to reduce the computational complexity. 
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In another variation, the first and the second passes are 
combined into a single pass in the following way. Starting 
from the top-left pixel of the low-resolution image, the val- 
ue of the pixel X 1 is first modified according to the teach- 
ings in the first pass of the preferred method. Next, the 
values of the pixels X 2 , X 3 , and X 4 are modified accord- 
ing to the STEP(S) 1-3 of the second pass. Scanning the 
image from left to right and from top to bottom, this pro- 
cedure is repeated for every pixel in the low-resolution 
image. The value of each pixel can be modified up to four 
times according to this procedure. However, since the 
order in which the adjustment takes place is different 
from the preferred method, in general, a different image 
will result. In yet another variation, the first pass is imple- 
mented as described in the preferred method, but the 
three steps in the second pass are implemented as three 
separate passes. That is, starting from the top-left pixel 
of the low-resolution image, the image is scanned from 
left to right and top to bottom and the STEP 1 of the sec- 
ond pass is applied to all the pixels. Then, the same pro- 
cedure is followed for STEP 2. Finally, STEP 3 is imple- 
mented in a similar fashion. Again, this results in a dif- 
ferent order in which the various pixels are processed. 
In yet another variation, after the execution of the first 
pass as described in the preferred method, the three 
steps in the second pass are carried out, but at each 
step, instead of adjusting the values of the pixels X 2 , X 3 , 
or X 4 , the value of the pixel X, is modified. Again, this 
will result in a different order of modifying the low-reso- 
lution pixel values than any of the schemes described 
before. 

Furthermore, although the invention has been de- 
scribed with a watermark insertion unit which introduces 
high contrast edges, the man skilled in the art would re- 
alize that, as already mentioned earlier, for some imag- 
es, the use of the invention provides an improvement 
over the prior art even when no watermark insertion unit 
is provided. 

Finally, while there has been shown what are con- 
sidered to be the preferred embodiments of the inven- 
tion, it will be manifest that many changes and modifica- 
tions may be made therein without departing from the 
essential spirit of the invention. It is intended, therefore, 
in the annexed claims, to cover all such changes and 
modifications as may fall within the true scope of the in- 
vention. 



Claims 

1. A method for reducing quantization artifacts in a 
hierarchical image system of the type that decom- 
poses a high resolution image into low resolution 
image components, including low resolution image 
pixel values, and residual image pixel values com- 
prising the steps of: 

a) determining the difference values between 



the pixel values in the low resolution image and 
the corresponding pixel values in the higher res- 
olution image; 

b) comparing each of the difference values 
5 against quantizer conversion values to deter- 
mine if an error would exist for an executed con- 
version; 

c) adjusting the image pixel values within the 
low resolution image component if an error 

10 would exist, to a value that eliminates the error 

to form an adjusted low resolution image; and 

d) using the unadjusted image pixel values as 
part of the adjusted low resolution image com- 
ponent if an error does not exist. 

15 

2. A method according to Claim 1 further comprising : 

e) forming a predicted higher resolution image 
by interpolating the pixel values of the adjusted 

20 low resolution image components of steps c) 

and d); 

f) determining the difference values between 
the interpolated pixel values in the predicted 
higher resolution image and the corresponding 

2S pixel values in the higher resolution image; 

g) comparing each of the difference values 
against quantizer conversion values to deter- 
mine if an error would exist for an executed con- 
version; 

h) adjusting further the image pixel values within 
the adjusted low resolution image component if 
an error would exist, to a value that eliminates 
the error to form an adjusted low resolution 
image; and 

35 i) using the unadjusted image pixel value as part 

of the adjusted low resolution image component 
if an error does not exist. 

3. A method for according to Claim 2, wherein a step 
40 of applying watermark pixel values to pixel positions 

within the low resolution image is performed before 
step a) and said modified low resolution image is 
used as low resolution image. 

45 4. A method according to Claim 3, wherein the pre- 
dicted higher resolution image of step e) is formed 
by sequentially interpolating the pixel values of a first 
block of low resolution image pixel values and steps 
f) through i) are repeated for each pixel in the higher 

so resolution image. 

5. A storage device having stored thereon the adjusted 
low resolution image components and the residual 
image pixel values of any of Claims 1 to 4. 

55 

6. Apparatus for reducing quantization artifacts in a 
hierarchical image system of the type that decom- 
poses a high resolution image into low resolution 



8 



15 EP 0 705 025 A2 16 

components, including low resolution image pixel 
values, and residual image pixel values, comprising: 

means for determining the difference values 
between the pixel values in the low resolution 5 
image and the corresponding pixel values in the 
higher resolution image; 
means tor comparing each of the difference val- 
ues against quantizer conversion values to 
determine if an error would exist for an executed to 
conversion; and 

means for adjusting the image pixel values 
within the low resolution image component if an 
error would exist, to a value that eliminates the 
error to form an adjusted low resolution image is 
component and using the image pixel value as 
part of the adjusted low resolution image com- 
ponent if an error does not exist. 

7. Apparatus according to Claim 6 further comprising : 20 

means for forming a predicted higher resolution 
image by interpolating the pixel values of the 
adjusted low resolution image component; 
means for determining the difference values 
between the interpolated pixel values in the pre- 
dicted higher resolution image and the corre- 
sponding pixel values in the higher resolution 
image; 

means for comparing each of the difference val- 30 
ues against quantizer conversion values to 
determine if an error would exist for an executed 
conversion; 

means for adjusting the image pixel values 
within the adjusted low resolution image com- 35 
ponent if an error would exist, to a value that 
eliminates the error to form a twice adjusted low 
resolution image component; and 
means for using the image pixel value as part 
of the twice adjusted low resolution image com- 40 
ponent if an error does not exist. 

8. Apparatus according to Claim 7, further comprising: 

means for applying pixel values to pixel posi- 45 
tions within the low resolution image to form a 
low resolution image component. 
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